Tremendous changes have been occurring in the Internet that influence our everyday lives. For example, online social networks have become the new meeting grounds. The development of such online social networks touch countless aspects of our everyday lives, providing instant access to people of similar mindsets, and enabling us to form partnerships with more people in more ways than ever before. Individuals may select to post information about themselves, their friends, family, or the like, on the Internet using any of a variety of tools, websites, or the like. In fact, because there are so many different places where a person might select to provide information, there are now just too many social sites out there for anyone to be able to keep up with all of the information that may be posted.
To help address this situation, many people employ aggregator tools. Aggregators are applications that are directed towards reducing the time and effort needed by a user to regularly check various websites for updates. The aggregator often performs such tasks, instead. Once a user subscribes to an aggregator, the aggregator checks various specified websites, or other sources, for changes in content, and if a change is detected, retrieves the update. The updates may then be provided to the subscriber, often in a single stream. One type of aggregator that has recently become popular is known as lifestreaming aggregators, or simply lifestreaming. Lifestreaming aggregators create an online record of a user's daily activities by aggregating their online content from such as blog posts, video logs (vlog) posts, online photo sites, and/or any of a variety of other specified social network sites. Users may provide their usernames for different sites, such as Flickr, YouTube, last.fm, Netflix, twitter, delicious, dig, or any of a variety of other sites (to name just a few). The lifestreaming aggregator then crawls the identified sites and aggregates or collects updates for the users.
The popularity of such lifestreaming may be attributed to that fact that such aggregation makes it easier for people to look at what someone else, such as friends, family, or the like is doing over the web in a convenient manner. A benefit for many users of such lifestreaming is that the user activity feeds are updated virtually in real-time providing enhanced user experience often over email feeds, or other slower mechanisms of communication. To make these aggregation feeds real-time, the aggregator often must crawl several websites aggressively. However, as their user base increases, many websites might find that they need to limit their hit rates for crawlers. This hit rate limiting is often done to prevent the website from responding too slow to other types of requests for content, and/or becoming overloaded and failing.
However, for many users of lifestreaming, real-time reporting may be essential, because information may become less useful as it gets older. Consider, for example, Bob posted a message on a website, or some status messaging site, stating that he would be “on Mission Street for the next hour—does anyone around want to have coffee?” If Alice, who was on Mission Street at that time received Bob's message outside of the specified one hour window, the update message would have virtually no value to her. Thus, it is with respect to these considerations and others that the present invention has been made.